34 research outputs found
From random walks to distances on unweighted graphs
Large unweighted directed graphs are commonly used to capture relations
between entities. A fundamental problem in the analysis of such networks is to
properly define the similarity or dissimilarity between any two vertices.
Despite the significance of this problem, statistical characterization of the
proposed metrics has been limited. We introduce and develop a class of
techniques for analyzing random walks on graphs using stochastic calculus.
Using these techniques we generalize results on the degeneracy of hitting times
and analyze a metric based on the Laplace transformed hitting time (LTHT). The
metric serves as a natural, provably well-behaved alternative to the expected
hitting time. We establish a general correspondence between hitting times of
the Brownian motion and analogous hitting times on the graph. We show that the
LTHT is consistent with respect to the underlying metric of a geometric graph,
preserves clustering tendency, and remains robust against random addition of
non-geometric edges. Tests on simulated and real-world data show that the LTHT
matches theoretical predictions and outperforms alternatives.Comment: To appear in NIPS 201
Likelihood-Based Diffusion Language Models
Despite a growing interest in diffusion-based language models, existing work
has not shown that these models can attain nontrivial likelihoods on standard
language modeling benchmarks. In this work, we take the first steps towards
closing the likelihood gap between autoregressive and diffusion-based language
models, with the goal of building and releasing a diffusion model which
outperforms a small but widely-known autoregressive model. We pursue this goal
through algorithmic improvements, scaling laws, and increased compute. On the
algorithmic front, we introduce several methodological improvements for the
maximum-likelihood training of diffusion language models. We then study scaling
laws for our diffusion models and find compute-optimal training regimes which
differ substantially from autoregressive models. Using our methods and scaling
analysis, we train and release Plaid 1B, a large diffusion language model which
outperforms GPT-2 124M in likelihood on benchmark datasets and generates fluent
samples in unconditional and zero-shot control settings
Benchmarking Multi-Domain Active Learning on Image Classification
Active learning aims to enhance model performance by strategically labeling
informative data points. While extensively studied, its effectiveness on
large-scale, real-world datasets remains underexplored. Existing research
primarily focuses on single-source data, ignoring the multi-domain nature of
real-world data. We introduce a multi-domain active learning benchmark to
bridge this gap. Our benchmark demonstrates that traditional single-domain
active learning strategies are often less effective than random selection in
multi-domain scenarios. We also introduce CLIP-GeoYFCC, a novel large-scale
image dataset built around geographical domains, in contrast to existing
genre-based domain datasets. Analysis on our benchmark shows that all
multi-domain strategies exhibit significant tradeoffs, with no strategy
outperforming across all datasets or all metrics, emphasizing the need for
future research
One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention
Recent works have empirically analyzed in-context learning and shown that
transformers trained on synthetic linear regression tasks can learn to
implement ridge regression, which is the Bayes-optimal predictor, given
sufficient capacity [Aky\"urek et al., 2023], while one-layer transformers with
linear self-attention and no MLP layer will learn to implement one step of
gradient descent (GD) on a least-squares linear regression objective [von
Oswald et al., 2022]. However, the theory behind these observations remains
poorly understood. We theoretically study transformers with a single layer of
linear self-attention, trained on synthetic noisy linear regression data.
First, we mathematically show that when the covariates are drawn from a
standard Gaussian distribution, the one-layer transformer which minimizes the
pre-training loss will implement a single step of GD on the least-squares
linear regression objective. Then, we find that changing the distribution of
the covariates and weight vector to a non-isotropic Gaussian distribution has a
strong impact on the learned algorithm: the global minimizer of the
pre-training loss now implements a single step of
GD. However, if only the distribution of the responses is changed, then this
does not have a large effect on the learned algorithm: even when the response
comes from a more general family of functions, the global
minimizer of the pre-training loss still implements a single step of GD on a
least-squares linear regression objective